What has an Artificial Neural Network (ANN) learned after being successfully trained to solve a task - the set of training items or the relations between them? This question is difficult to answer for modern applied ANNs because of their enormous size and complexity. Therefore, here we consider a low-dimensional network and a simple task, i.e., the network has to reproduce a set of training items identically. We construct the family of solutions analytically and use standard learning algorithms to obtain numerical solutions. These numerical solutions differ depending on the optimization algorithm and the weight initialization and are shown to be particular members of the family of analytical solutions. In this simple setting, we observe that the general structure of the network weights represents the training set's symmetry group, i.e., the relations between training items. As a consequence, linear networks generalize, i.e., reproduce items that were not part of the training set but are consistent with the symmetry of the training set. In contrast, non-linear networks tend to learn individual training items and show associative memory. At the same time, their ability to generalize is limited. A higher degree of generalization is obtained for networks whose activation function contains a linear regime, such as tanh. Our results suggest ANN's ability to generalize - instead of learning items - could be improved by generating a sufficiently big set of elementary operations to represent relations and strongly depends on the applied non-linearity.

"The brain is very busy when we sleep, repeating what we have learned during the day," said Maxim Bazhenov, PhD, professor of medicine and a sleep researcher at University of California San Diego School of Medicine. "Sleep helps reorganize memories and presents them in the most efficient way." In previous published work, Bazhenov and colleagues have reported how sleep builds rational memory, the ability to remember arbitrary or indirect associations between objects, people or events, and protects against forgetting old memories. Artificial neural networks leverage the architecture of the human brain to improve numerous technologies and systems, from basic science and medicine to finance and social media. In some ways, they have achieved superhuman performance, such as computational speed, but they fail in one key aspect: When artificial neural networks learn sequentially, new information overwrites previous information, a phenomenon called catastrophic forgetting.

The learning ability of artificial neural networks ("ANNs") falls under the scientific area of machine learning. Machine learning is a generic term for the artificial generation of knowledge from experience. More specific, an ANN learns from historical examples and can generalize these after the learning phase by learning the patterns contained in the examples. In machine learning, there are three learning paradigms. These include supervised and unsupervised learning as well as reinforced learning.¹

This article is a part of Artificial Neural Networks Serial, which you can check out here. In the previous blog posts, we covered some very interesting topics regarding Artificial Neural Networks (ANN). The basic structure of Artificial Neural Networks was presented, as well as some of the most commonly used activation functions. Nevertheless, we still haven't mentioned the most important aspect of the Artificial Neural Networks – learning. The biggest power of these systems is that they can be familiarized with some kind of problem in the process of training and are later able to solve problems of the same class – just like humans do!

Ever wanted to see your holiday snaps in the style of Van Gogh, or have your portrait painted by Picasso? Researchers have revealed an artificial intelligence algorithm that can learn to paint in the style of any artist - and repaint any snap you feed it. Researchers fed their system a series of old masters - and it turned a modern day snap into perfect pictures in the style of some of the world's best known paintings. The team say it can learn the style of any artist simply by analysing a single picture. The original image the team fed the algorithm this image of houses to be'reimagined' in the style of different artists'In fine art, especially painting, humans have mastered the skill to create unique visual experiences through composing a complex interplay between the content and style of an image,' the researchers from the University of Tubingen wrote in the journal Arxiv.